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Abstract — This paper presents an analysis of the concept of 
capacity for noisy computations, i.e. algorithms implemented by 
unreliable computing devices (e.g. noisy Turing Machines). The 
capacity of a noisy computation is defined and justified by 
companion coding theorems. Under some constraints on the 
encoding process, capacity is the upper bound of input rates 
allowing reliable computation, i.e. decodability of noisy outputs 
into expected outputs. A model of noisy computation of a perfect 
function / thanks to an unreliable device F is given together 
with a model of reliable computation based on input encoding 
and output decoding. A coding lemma (extending the Feinstein's 
theorem to noisy computations), a joint source-computation 
coding theorem and its converse are proved. They apply if the 
input source, the function /, the noisy device F and the cascade 
f~ x F induce AMS and ergodic one-sided random processes. 

I. Introduction and related works 

Reliable computation with unreliable devices, or in the 
presence of noise, has been the subject of numerous works 
within the vast field of fault- tolerant computing. Computa- 
tion can be made reliable using information and compo- 
nent/gate redundancy. Some works aim at identifying theoreti- 
cal "boundaries" on the amount of necessary and/or sufficient 
redundancy to achieve reliability. Recent references (see for 
example, [fl~|, 10, J3), H) continue to extend the stream 
opened by Von Neumann's seminal paper [5|. These works 
identify bounds (e.g., depth and size of circuits) and propose 
frameworks to design reliable computations mainly thanks to 
gate redundancy. These papers do not address the question 
of boundaries about information redundancy. This subject, 
through the concept of capacity and coding theorems, has been 
thoroughly studied for data communication. It has not been 
the case for computation although the two problems (noisy 
computation and noisy communication) are very close: the 
matter is to retrieve expected values from the outputs of a 
random process. 

The question of whether a noisy computation possesses a 
capacity (or equivalently whether some coding theorems for 
noisy computation hold and in which cases) has been raised 
by R Elias in 1958 0. There is a practical consequence 
in answering positively this question. This would mean that, 
given a noisy implementation of an expected (e.g., Turing 
computable) function, it is possible to find families of efficient 
input codes which asymptotically allow an almost perfect 
computation. Efficiency means having an input encoding rate 
which could remain strictly positive or arbitrarily close to a 
capacity when the length of the code tends to infinity. 

In a strongly constrained context (independent encoding of 



operands for bit-by-bit boolean operations), P. Elias obtained 
negative first results on the existence of a noisy computation 
capacity ( J6|) . This work was deepened by Peterson and 
Rabin in Q and by Winograd in [8|. One of the major 
conclusions of these studies was that reliable computation with 
positive rate (the ratio - of encoding fc-length input block in 
n-length blocks of binary symbols) in the presence of noise is 
not possible for some boolean operations (e.g., AND) under 
some assumptions (independent coding of operands, bijective 
decoding and bit-by-bit operation). This led to the conclusion 
that, under these assumptions, there is no capacity for such 
noisy operations. It is worth noting that the assumptions 
were made to forbid the reliable encoder and decoder to 
"participate" to the computation of the expected operation. 

In (9), Ahlswede went into the subject in greater depth 
and made an important contribution in characterizing contexts 
in which a capacity for noisy computations cannot exist. It 
appears that the characteristics of the decoding function play 
a fundamental role. If the inverse of the decoding function 
is injective and monotonic then weak converse theorems hold 
for the average and maximal error probabilities. If, in addition, 
the inverse of the decoding function preserves the logical AND 
(this implies monotonicity), then strong converses hold. These 
theorems state that the rate of encoding tends to when the 
block code length tends to infinity. The hypotheses made in 
|6|-[8|, i.e., independent encoding of operands and bijective 
decoding, imply monotonicity of the inverse of the decoding 
function. On this aspect, [9| supersedes [6|-|8|. 

Nevertheless, these negative results do not imply the abso- 
lute impossibility to identify a capacity for noisy computation. 
They characterize codes, encoding and decoding processes 
which cannot open this ability. To define a capacity for noisy 
computations, assumptions must be relaxed. 

To the author's best knowledge, the first positive answer 
given through a definition of a capacity of a noisy computation 
(in fact similar to the one for a noisy channel) and a coding 
theorem came from Winograd and Cowan in llTOl . In [10|, the 
entropy H(X\F(X)) of the input source conditioned by the 
noisy computation output is assessed as a noise measure. As it 
is the equivocation between the noisy output and the input, this 
quantity is not relevant, in full generality, as the equivocation 
due to the sole noise: it encompasses also the amount of 
information lost by computation. But, in a special case of 
noisy functions called decomposable modules, H(X\F(X)) 
actually measures the equivocation due to noise. Decompos- 
able modules are noisy functions which can be modeled by a 



perfect function followed by a noisy communication channel: 
the error probability depends on the desired output value rather 
than on the input value. These peculiar functions, though 
noisy, make the context equivalent to that where the reliable 
encoder computes and encodes the expected function result 
before communication through a noisy channel. Due to the 
restriction of considering decomposable modules, [10] did not 
completely succeed in proposing a noisy computation capacity 
in a general scope ( [10], theorem 6.3, pages 47-48 ). 

Noisy computation capacity is also considered in reliable 
reconstruction of a function of sources over a multiple access 
channel. Much more recently, a definition of noisy compu- 
tation capacity is established by Nazer and Gastpar in QT] 
and is totally consistent with the one proposed here. Nazer 
and Gastpar demonstrate the possible advantages of joint 
source-channel coding of multiple sources over a separation- 
based scheme, allowing a decoder to retrieve a value which 
is a function of input sources. This context makes relevant 
the proposed distributed encoding process which perfectly 
performs a computation equivalent to the desired function. The 
encoder outputs are then transmitted through a noisy MAC to 
a decoder (see proofs of Theorems 1 and 2 of ifTTI ). This also 
models a noisy computation as a perfect computation followed 
by a noisy transmission of the result. It can be noticed that 
[ 1 1 and [ 1 1 ] relax the assumptions of [6 1-[9] in a similar way: 
all goes as if the operands are jointly coded into an encoded 
form of the expected function result before being handled by 
a noisy communication channel. 

The present paper establishes a model setting down the 
problem of noisy computation (section |H}, a definition of the 
capacity of a noisy computation with respect to an expected 
function and a coding lemma ( section [TTTb- A model for reliable 
computation is given, section [TV] Based on this model, a 
joint source-computation coding theorem and its converse are 
stated and proved in Section [V] This theorem aims at formally 
capturing practical approaches in which reliable computation 
of a function g is obtained thanks to a noisy apparatus F 
computing with noise a function / (e.g., a regular arithmetic 
addition g obtained from the noisy actual circuit F imple- 
menting / which is an addition acting on residue encoded 
operands, lfl2l ). The input source, /, g, F and the cascade 
are supposed to be AMS and ergodic one-sided random 
processes or channels, extending 0131 to more general random 
processes and algorithms. The perfect function / is assumed 
unary (as is a Turing computable function), n-ary functions 
can be modeled as unary ones by concatenating n input values 
in one "meta"-input and thus modeling a joint coding of 
operands. This relaxes the assumptions of [6|-[9|. 

II. Model for Noisy Computation 

In this section, the notations used follow [14] and cover 
countable alphabets, assumed standard (thus conditional prob- 
abilities are regular). 

Let X = {Xi\i G 1} a random process with values in 
(A 1 . B A x) where A is a countable alphabet and I a countable 
set of indexes (e.g., N). B A i denotes the er-field generated by 



the rectangles (chapter 1, [14]). The random process X is the 
source of inputs of the noisy computation. 

The noisy computing device is modeled as a random chan- 
nel, i.e. a set of conditional probabilities F = {F. x , x G A 1 }, 
taking X as input and producing as an output a random process 
Z = {Zi;i G 1} on (C z , Bex) where C is a countable 
alphabet. The hookup Pxz = PxF is the probability measure 
characterizing the actual noisy computation with an input flow 
represented by X. From lfT4l . chapter 2, VO G B a i xC i\ 

Pxz{0)= ( P z \x(O x \x)dP x = f P x \z(O z \z)dP z 

where O x — {z G C x /(x,z) G O}. The probabilities 
{Pz\x{-\ x ), x G A x } defines the channel X -)• Z (F x = 
Pz\x{-\ x )) and {Px\z(-\ z ), z 6 C 1 } tne "reverse" chan- 
nel F~ l . Pxz fully determines the channels F and F -1 . 
Conversely, if X and a set of conditional probabilities 
{Pz\x(-\ x ): x £ A 1 } (i.e. F) are given, then Pxz and the 
output process Z are well defined. A functional notation 
Z = F(X) will be used below. 

The desired (i.e. perfect) computation will be represented by 
a measurable function / : A 1 —> B x where B is a countable 
alphabet. Y = f(X) is a random process of distribution 
Py = Pxf 1 - The function / defines a deterministic channel 
X — > Y = f{X) which is a set of conditional probabilities 
{Pf(X)\x(-\x),x G A 1 } (See [14], chap. 2): 

VG G B B x,P Kx) \ x (G\x) = l/-i( G) (z) Px a.e. (1) 

F and / determine a channel f(X) — > Z which is a 
cascade of the reverse channel / _1 followed by F (figure ([TJ. 
The noisy computation model should not be understood as 

/ 



(A I ,B A x,P x ) (B x ,B B z,P Y ) 
(C x ,B c i,P z ) 

Fig. 1. Model for Noisy Computation 

a cascade made of a perfect function followed by a noisy 
channel (as done in ifTOll for instance). The cascade f~ x F 
is an "artifact" on which a channel coding theorem will be 
invoked to build a code for the noisy computation. 

From the expression giving the probabilities of a cascade 
of channels ( 04], chap. 2) and applying (fTJ, it easily comes 

that P Y \z = Px\zr X - 

If / is bijective and A — B, the model is the noisy channel 
one. 

III. Capacity of a Noisy Computation 

The alphabets are now assumed finite and the er-fields B A , 
B B and B c are the sets of subsets of A, B and C. f n {X n ) 
stands for the random value Y n (the n first symbols of Y), 
F n {X n ) for Z n . [A x B, Xf; AxC, XF] denotes the noisy 



computation F of / on X. X, F, f and the cascade f~ 1 F 
are assumed AMS and ergodic. Thus entropy rates are limits 
and ergodic theorems hold ( lfl4l '). 

The Feinstein's theorem is reminded ( [14], chap. 12): 
Theorem 1 (Feinstein's theorem): Let [A x B,fii>] be an 
AMS and ergodic hookup of a source fi and channel v. 
Let I ^ = I^ U (X;Y) denote the average mutual informa- 
tion rate and assume that is finite (this is the case 
if the alphabets are finite.)- Then for any R < I ^ and 
for any e > 0, there exists, for n large enough, a code 
{(oji, Ti) G A n x B B ™ , i = 1, ■ • • , M} such that M = [e nR \ 
and Mi = 1, • • • , M, v n {T1\uji) < e 

v n is the channel induced by the source /i, ( IfBll . chap. 12). 

The following definition introduces the typical input rate 
which will be shown to be the rate at which a source should 
produce typical inputs for a random process to allow to recover 
the desired function results by decoding. 

Definition 1: The typical input rate of the source X for the 
noisy computation F with respect to the perfect function / is 
the following limit, denoted B(X,f,F): 

H(X n \f n (X n )) + I(F n (X n ); f n (X n )) 
n 

Taking into account the hypothesis ( AMS and ergodic source 
and channels), B(X, f, F) is well defined. Simple algebra 
gives B{X,f,F) =H(X) -H(f(X)\F(X)) 

Definition 2: Let [A x B, Xf; A x C, XF] be a noisy com- 
putation on finite alphabets A, B and C. A [M, n, e]-Feinstein 
code for the noisy computation [A x B, Xf; Ax C, XF] is a 
set {(Af,Tf) G B A ™ x B c «,i = !,■■■ ,M] such that: 

!) p F{x)\x( T 1\ xn ) < e for an y x " e A ?- £ = 1, • • • i M 

2) Vi = 1, • • ■ , Af, 3y 2 G B" such that A™ = (/")-%) 
F n (X n ) e-reliably computes f n (X n ) on the code 
{(A?,T?),i = l,..- ,M}. 

Lemma 1 (Feinstein's theorem for Noisy Computation): 
Let [A x B, Xf; AxC, XF] be a noisy computation on finite 
alphabets. For any R < B(X, f, F), for any e > 0, n large 
enough, there exists a [\_e n ( R ~ H ( x v( x ^ },n, e]-Feinstein 
code for F™(X") to e-reliably computes f n (X n ). 

Proof: R < B(X,f,F) => R' = R - ~H(X\f(X)) < 
I(f(X),F(X)). Then, thanks to the Feinstein's theorem, 
since Pf(x).F(x) is AMS and ergodic (by assumption), for 
n large enough, there exists a Feinstein code {(yi,Ti) G 
B n x B c ™;i = l,-- - ,M} such that M = [e nR ' \ and 
Vi = l,--- ,M;Pg (x)15 (r?| W ) <e 

Let x belong to (/") 1 (y). Considering the cascade X n 
f n (X n ) -4 F"(X n ), for any A; = 1, • ■ • , M: 

Phx)\xin\x) = J P^ xmx) (n\y)dP^ x) \ x {y\x) 

Phx)\xin\x) = I Phx)\f(x){n\y)dP)\ x) \ X {y\x) 

J {Vk} 

+ I pF(x)\f(x)(n\y)dP f (x)\x(y\x) 

J {vkV 

< Pf(x)\x ({y k } \x) .e + P; (x)|x ({y k } c \x) 



If x € (/ n )- x (l/fc) then Pf {x)]x ({y k } \x) = 1 
and Pf (x) \ x {{VkY \x) = 0, hence Va; G f^ 1 {yk), 

We can conclude this section by the definition of the typical 
input capacity of a noisy computation. 

Definition 3: The typical input capacity of the noisy func- 
tion F with respect to the perfect function / is Cf(F) = 
su Pams em p x f, F), the supremum is over all AMS and 

ergodic sources X. 

The equivalent expression Cf{F) = 
sup AMSergPx \H(X) - H(f(X)\F(X))] shows that this 
capacity boils down to the "usual" channel capacity when / 
is a bijection, in which case H(f(X)\F{X)) = H(X\F(X)). 

IV. Reliable computation 

There is a need ( J6), |£Sj ) to constrain the encoding and 
decoding processes to avoid the following cases: 

• either an (assumed perfect) encoder which computes the 
expected function, encodes the result before transmission 
through the random process (considered as a noisy trans- 
mission channel) 

• or an encoder which encodes input values for reliable 
transmission through the random process (considered 
here also as a noisy transmission channel) and a decoder 
(assumed also reliable) which decodes (almost perfectly) 
and computes (perfectly) the expected function. 

Considering that a computation g is a "true" computation 
if the entropy is reduced (E(X']g(X')) > H(g(X')) < 
H(X') (else it is communication), the model must be tar- 
geted to be mainly relevant for non-injective functions (i.e. 
H(X'\g(X')) > 0). For injective g, this becomes the classical 
reliable transmission model. With the constraint that both 
encoding and decoding are based on injections (in a sense 
made precise below) then the encoder and the decoder cannot 
compute (at least totally) the desired function as they do not 
reduce entropy. 

The proposed model of the complete process to reliably 
compute a function g : A' 1 — > B lT acting on a source X' , 
thanks to a noisy implementation F of a function / : A 1 — > 
C 1 is the following: 

• encoding: let X n be the n th extension of an source 
for which we have a maximal code (Af, 1^=1,... t M 
allowing to e-reliably compute f "(X n ) by F n {X n ) (cf 
lemma Q] and definition |2]i ; a typical fc-sequence x' of 
X' k is encoded into a typical given yi (this important 
assumption is discussed in the conclusion) n-sequence of 
X n by a injective function, say U, such that U(x') G Af 
for some i = 1 , • • • , M 

• computation of the noisy function: F n is applied to 
U(x') producing a typical rt-sequence F n (U(x')) of 
F n {X n ) where F n (U(x')) belongs to a given T t (with 
probability greater than 1 — e) 

• decoding: the first step is to associate to F n (U(x')) 
the typical n-sequence yt of /™(X") corresponding 
to Ti, the second step is to apply to y.- L a function 



V : {yi, . . . ,yivi} {typical k-sequences of g k (X' k )} 
such that V(yi) = g k (x') 
A decoding error occurs when one obtains a n-sequence yj 

(or equivalently a Tj) such that g k (x') = V(yj) ^ g k (x') 

To be able to define a decoding function V (i.e, a determin- 
istic decoding), the encoding function U has to be such that 
the typical (given yi) n-sequences of one A" = 
(Ui 6 {yi > • • • > 2Aa/ }) are use d for encoding typical fc-sequences 
of only one (g k )^ 1 (z), z typical fc-sequence of g k {X' k ). 

We also require that V be an injection (as we have required 
from U). 

The typical fc-sequences of a (g k )~ 1 (z), z typical fc- 
sequence of g k (X' k ), are encoded in typical (given j/j) n- 
sequences of one and only one A™ . So, if x[ and x' 2 are two 
typical fc-sequences of X' k : 

f n (U(x[)) = P(U(x' 2 )) o = 

The model fulfills the constraints identified above. The 
encoder implements an injection and thus cannot compute the 
desired function / nor g (if / and g are not injective). The 
same comment applies to the injective decoding step V. 

V. A CODING THEOREM AND ITS CONVERSE 

The sources, functions, noisy function and the cascade 
f~ 1 F are assumed AMS and ergodic. 

Definition 4: With the notations of section [V] the ratio 



R 



k.H(X') 



is called the typical encoding input rate. A 



rate R is said to be achievable with respect to the function 
/ if there exists a sequence of codes of size n such that the 
maximal probability of decoding error tends to as n tends 
to infinity. 

Theorem 2: If R < Ct(F), then R is achievable w.r.t /. 
Proof: 

This proof, although identical to that in |[T3l . is given as 
it includes the starting point for the proof of the converse 
theorem. First, it is shown that the injective encoding of 
typical fc-sequences of a set (g k )~ 1 (z) on typical (given yi) 
n-sequences belonging to A™ is possible for suitably chosen fc 
and n (lossless coding). Secondly, it is shown that, at encoding 
input rates below capacity and for fc and n suitably chosen, 
the sets A" are almost as many as the sets (g k )~ 1 (z). 

Let 5" > 0. Since Q is dense in R, there exist fc and n such 

that: _ _ 

H{X'\g{X')) ^ n ^ H{JC\g{X')) + 5" 

H(X\f(X)) fc H(X\f(X)) 
Moreover, fc and n can be chosen as large as needed. Thus: 

k.H(X'\g('X)) k.(H(X'\g(X')) + S") 

H(X\f(X)) ' H(X\f(X)) 

We can choose 5,6' > and < e < 1/2 small enough for: 
k.(H(X'\g(X')) + 6)-log(l-2e) 



H(X\f(X))-6> 



< n 



< 



k.(H(X'\g{X')) + 6 + 5" 
H(X\f(X))+S> 



giving 

k.(H{X'\g(X')) + 5) < log(l-2e) + n.(H(X\f(X))~S') 
< n.{H{X\f{X)) + 5') < k.(H(X'\g(X')) +6 + 5") 

If i>i is the number of typical fc-sequences of (g k )~ 1 (z) and 
v 2 is the number of typical (given y.j) n-sequences in an A", 
we have (by conditional AEP): 



v\ < e 



k.(H(X'\g(X'))+S) 



< (l-2e)e 



n.{H(X\f(X))-5') 



<V 2 < e "-( ff ( x l/( x ))+' 5 ') < e k.{B{X'\g(X'))+&+S") 



It is thus possible to find an injection from the set of typical 
fc-sequences of (g k )^ 1 (z) on the subset of typical sequences 
(given y^ of A™. This shows the first step. 
_Assume that R = kH(X')/n < H(X) - 
H(f(X)\F(X)) < C f {F). Such a X exists by definition of 
C f (F). So 

k(H(g(Xy) + H(X'\g{X')))< 

n.(H(f(X)) - H(f(X)\F(X))) + n.H(X\f(X)) 

By ©, n.H(X\f(X)) - k.H(X'\g[X')) < k.S" thus 
kH(g(X')) < n.(H(f(X)) - H(f(X)\F(X))) + k.S" 
ei, 5"' > can be chosen small enough in order to get: 

e k(H(g(X'))+S"') < e n.(H(f(X))-HU(X)\F(X)) + %.6"-e 1 ) 

If v-i is the number of typical fc-sequences of g k (X' k ) and 
M is the size of the code (i.e., the number of (A",I\)), we 
have (by AEP and Lemma [T): 



v-i < e 



k(H(g(X'))+8'") 



< e 



n.(H(f(X))-H(f(X)\F(X)) + %.5"-e 1 ) < M 



The assumed model, by the constraints on encoding, implies 
that the best ratio (i.e., the smaller) j of encoding respects 

the inequality ©: < ? < "^}^',]}+ 5 " . Let 



H(X\f(X)) 



H(X\f(X)) 



7 = ^=L — ^ — -ii. To respect the encoding constraints (typical 

H ( x \f( x )) 

sequences are "injectively" encoded into typical sequences), a 
rate R = ^H(X') must be such that R < 

Theorem 3: If R > Cf(F), there is no code such that the 
error probability tends to as n — s> oo 

Proof: The decoding is deterministic then: 

1) F n {X n ) -» f n (X n ) -» g k (X' k ) is a Markov Chain 
thus g k (X' k ) -» f n (X n ) F n {X n ) is a Markov 
Chain 

2) f n (X n ) -> F n (X n ) g k {X rk ) is a Markov Chain 



Hence g k {X lk ) -> f n (X n ) -> F n (X n ) -> g k {X' k ) is a 
Markov Chain. This implies that, Vn, k such that ^ < —: 

I(g k (X' k y i9 HX^)) < I(f n (X n );F n (X n )) hence 

H{g\X' k )) - H(g k (X' k )\g k ^ k )) < 
I(f n (X n );F n (X n )) 

=► #pC' fe ) - H(X' fc |ff*(JT /fc )) - H{g k (X' k )\g k {X Jk )) < 

I(f n (X n );F n (X n j) 

by Fano's inequality: 

H{X lk ) - H(X' k \g k (X' k )) 

-(H 2 (P e (k)) + k.P e (k).log(\B'\)) 

< I(f n (X n );F n (X n )) 

H{X*) H 2 (P e (k)) 
=> 1 J, Pe(k).log(\B |) < 

H{X' k \g k (X' k )) I(f n (X n );F n (X n )) 



H(X' k ) H 2 (P e (k)) 
k k 
H(X' k \g k {X' k )) 



P e {k).log(\B'\) < 
I(f n (X n );F n (X n )) 



k ^ n 

If the error probability is asymptotically (i.e., 
linik^oo{Pe(k) = 0) then necessarily (letting k and n tend 
to infinity): H(X')_^ H(X'\g(X')) +j.I(f(X); F(X)). But 
H(X'\g(X')) = jH(X\f(X)), then: 

H(X') < 7 . (H(X\f(X)) + I(f(X);F(X))) 

since R < — - — '-, we obtain 

— 7 

R < H(X\f(X)) + I(f(X); F(X)) < C f (F) 
Thus if R > Cf(F) then the error probability does not vanish. 



VI. Discussion and conclusion 

The coding lemma, the coding theorem and its converse 
assume that the sources (X and X'), the channels {F, /, g) 
and the cascade f~ 1 F are AMS and ergodic. Cases can be 
identified where it is possible to derive the AMS property and 
ergodicity of f F from properties of X, F and / (e.g., if X, 
F are stationary and weakly mixing and / AMS and ergodic 
then f~ 1 F is AMS and ergodic). Due to lack of space, this 
question is not addressed here, neither the identification of 
classes of AMS and ergodic functions / and g. 

The model of reliable computation assumes that the encoder 
and the decoder are perfectly reliable. This assumption could 
be justified by quoting from J8) "The computation system 
[model] was devised for the sole purpose of studying the 
relation of information theory of reliable automata" . Moreover 
we could argue in addition that if the complexity of the 
computation device is of a much greater magnitude than that of 
the encoder and decoder then the unreliability of the encoder 
and decoder have almost no impact on the overall reliability 



of the computation and thus can be neglected. For complex 
systems (e.g., based on significant software volume), this is 
quite realistic. In any case, it is impossible to overcome the 
fact that the reliability reached is at the best the reliability 
of the final decoding device. The only way is to built a 
intrinsically reliable enough decoder (for example thanks to 
gate redundancy). A noisy encoder is a noisy computation 
itself and thus can be handled from the point of view of 
"cascaded noisy computations". This is outside of the scope 
of the present paper. 

The proposed model of reliable computation involves two 
perfect functions g and /. This is intended to capture major 
real cases as already mentioned. Another motivation to use an 
"ancillary" function / in the model is that this is an efficient 
way to define an input code, meaning a family of subsets 
(Af)i—± t ... that do not overlap and whose "images" by the 
noisy function do not overlap "too much" (i.e., fall into disjoint 
Ti with high probability). Defining such family is defining 
(partially) a function / by picking, for each i an yi and stating 
f (Vi) = Ai.ln addition / allows a characterization of a kind 
of size of the sets {f n )^ 1 {yi) through the conditional entropy 
rate H(X\f(X)). This motivates also the constraint of coding 
by conditionally typical sequences. While the sets 
are "balanced", for large n, with respect to the number of 
(conditionally) typical sequences they contain, their cardinali- 
ties might be very different and bounds are not straightforward 
to obtain. Thus, the use of all possible elements of (f n )~ 1 (yi) 
forbids to characterize all the {f n )^ 1 {yi) by the same number 
measuring the "encoding" power. The same difficulty forbids 
to state a converse as well. The "encoding by conditionally 
typical sequences" trick overcomes this difficulty. 
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